READ - I've created a python script to allow the user to click the button and then they can either have all of the underyling code shown, OR they can just look at the raw output (charts, plots, whatever).
As you know, sometimes these notebooks contain a fair amount of code... and sometimes folks just want the results... here is an example

Code defaults to NOT showing any code, so click the toggle button to view the show the underlying code...

Unsupervised Machine Learning Approach

Methodology: DBSCAN

Univariate Temperature Data

Import standard python libraries

Move directories to read in file

Read in data

Plot snipper for understanding

Show number of unique sensors and sensor parameters being recorded:

Grab window in 2019

We will grab a particular window in 2019 of just over two months.

We still have a fair amount of data (given sensors record every 25s, and we have many many sensors at this particular location)

Filter by parameter of temperature

Filter to on a single sensor for now (temp only parameter)

Start and end of data:

Will we successfully capture the polar vortex that hit Chicago in 2019 ?

Looking at our sensor temperatures

Converting from C to F

image.png

Saving some plots

Plotting our raw temperature data to see what it looks like

https://goo.gl/maps/fjpnRfH7j4BkUfbR9

DBSCAN fundamentals

Help on class DBSCAN in module sklearn.cluster._dbscan:

class DBSCAN(sklearn.base.ClusterMixin, sklearn.base.BaseEstimator)

   DBSCAN(eps=0.5, *, min_samples=5, metric='euclidean', metric_params=None, algorithm='auto', leaf_size=30, p=None, n_jobs=None)

   Perform DBSCAN clustering from vector array or distance matrix.

   DBSCAN - Density-Based Spatial Clustering of Applications with Noise.
   Finds core samples of high density and expands clusters from them.
   Good for data which contains clusters of similar density.

   Parameters
   ----------
   eps : float, default=0.5
       The maximum distance between two samples for one to be considered
       as in the neighborhood of the other. This is not a maximum bound
       on the distances of points within a cluster. This is the most
       important DBSCAN parameter to choose appropriately for your data set
       and distance function.

   min_samples : int, default=5
       The number of samples (or total weight) in a neighborhood for a point
       to be considered as a core point. This includes the point itself.

   metric : string, or callable, default='euclidean'
       The metric to use when calculating distance between instances in a
       feature array. If metric is a string or callable, it must be one of
       the options allowed by :func:`sklearn.metrics.pairwise_distances` for
       its metric parameter.
       If metric is "precomputed", X is assumed to be a distance matrix and
       must be square. X may be a :term:`Glossary <sparse graph>`, in which
       case only "nonzero" elements may be considered neighbors for DBSCAN.

       .. versionadded:: 0.17
          metric *precomputed* to accept precomputed sparse matrix.

   metric_params : dict, default=None
       Additional keyword arguments for the metric function.

       .. versionadded:: 0.19

   algorithm : {'auto', 'ball_tree', 'kd_tree', 'brute'}, default='auto'
       The algorithm to be used by the NearestNeighbors module
       to compute pointwise distances and find nearest neighbors.
       See NearestNeighbors module documentation for details.

   leaf_size : int, default=30
       Leaf size passed to BallTree or cKDTree. This can affect the speed
       of the construction and query, as well as the memory required
       to store the tree. The optimal value depends
       on the nature of the problem.

   p : float, default=None
       The power of the Minkowski metric to be used to calculate distance
       between points. If None, then ``p=2`` (equivalent to the Euclidean
       distance).

   Attributes
   ----------
   core_sample_indices_ : ndarray of shape (n_core_samples,)
       Indices of core samples.

   components_ : ndarray of shape (n_core_samples, n_features)
       Copy of each core sample found by training.

   labels_ : ndarray of shape (n_samples)
       Cluster labels for each point in the dataset given to fit().
       Noisy samples are given the label -1.

   Examples
   --------
   >>> from sklearn.cluster import DBSCAN
   >>> import numpy as np
   >>> X = np.array([[1, 2], [2, 2], [2, 3],
   ...               [8, 7], [8, 8], [25, 80]])
   >>> clustering = DBSCAN(eps=3, min_samples=2).fit(X)
   >>> clustering.labels_
   array([ 0,  0,  0,  1,  1, -1])
   >>> clustering
   DBSCAN(eps=3, min_samples=2)

   See Also
   --------
   OPTICS : A similar clustering at multiple values of eps. Our implementation
       is optimized for memory usage.

   Method resolution order:
       DBSCAN
       sklearn.base.ClusterMixin
       sklearn.base.BaseEstimator
       builtins.object

   fit(self, X, y=None, sample_weight=None)
       Perform DBSCAN clustering from features, or distance matrix.

       Parameters
       ----------
       X : {array-like, sparse matrix} of shape (n_samples, n_features), or             
           (n_samples, n_samples)
           Training instances to cluster, or distances between instances if
           ``metric='precomputed'``. If a sparse matrix is provided, it will
           be converted into a sparse ``csr_matrix``.

       sample_weight : array-like of shape (n_samples,), default=None
           Weight of each sample, such that a sample with a weight of at least
           ``min_samples`` is by itself a core sample; a sample with a
           negative weight may inhibit its eps-neighbor from being core.
           Note that weights are absolute, and default to 1.

       y : Ignored
           Not used, present here for API consistency by convention.

Converting raw data via StandardScaler()

-- ITERATION 1 -- Running DBSCAN on the raw data to determine clusters of time series parameter values

-- ITERATION 1 -- Plotting the clusters that DBSCAN identified, with our base eps and min_samples hyperparameters:

-- ITERATION 1 -- Translating / Understanding what we are seeing with anomaly detection:

-- ITERATION 1 -- Visualizing some closeup ranges (red anomaly and blue anomaly regions):

As a reference, we use plotly to show all the temperature values over the entire range, so readers can home in on the window they wish to view/explore...

https://ghcdn.rawgit.org/tombresee/SensorAnalysis/main/ENTER/results/temp_1q_2019_chicago.html

-- ITERATION 1 -- Outputting and exporting the actual anomalous values (ranges), and plotting shaded regions...



Appendix: Ignore All This

Using Tuning Approach:

from sklearn.cluster import DBSCAN
clustering1 = DBSCAN(eps=0.09, min_samples=6).fit(np.array(ts_dataframe['Normalized Profit']).reshape(-1,1))

labels = clustering1.labels_

outlier_pos = np.where(labels == -1)[0]

x = []; y = [];
for pos in outlier_pos:
    x.append(np.array(ts_dataframe['Normalized Profit'])[pos])
    y.append(ts_dataframe['Normalized Profit'].index[pos])

plt.plot(ts_dataframe['Normalized Profit'].loc[ts_dataframe['Normalized Profit'].index], 'k-')
plt.plot(y,x,'r*', markersize=8)  
plt.legend(['Actual', 'Anomaly Detected'])
plt.xlabel('Time Period')
plt.xticks([0, 20, 40, 60, 80, 99],[ts_dataframe.index[0],ts_dataframe.index[20], ts_dataframe.index[40], ts_dataframe.index[60], ts_dataframe.index[80], ts_dataframe.index[99]] ,rotation=45)
plt.ylabel('Normalized Profit')

RESULTS:
========    

db = DBSCAN(eps=0.03, min_samples=6, metric='euclidean', n_jobs=-1)
 0    139811
 3      4950
 2        37
 1        26
-1        19
 4        10


db = DBSCAN(eps=0.04, min_samples=5, metric='euclidean', n_jobs=-1)
 0    144840
-1        13


db = DBSCAN(eps=0.04, min_samples=7, metric='euclidean', n_jobs=-1)
 0    139848
 1      4989
-1        16


db = DBSCAN(eps=0.04, min_samples=7, metric='euclidean', n_jobs=-1)
 0    139848
 1      4989
-1        16


db = DBSCAN(eps=0.1, min_samples=10, metric='euclidean', n_jobs=-1)
 0    198364
 4      4990
 1        66
-1        41
 2        10
 3        10




``